Q-learning-based optimal digital feedback control with computation time delay of linear discrete-time systems

In embedded computers, there are delays due to computation time. Unless they are considered, a controlled system may be unstable. If the system is unknown, Q-learningbased optimal control is one of the useful approaches. Applying it to a system, we can obtain the optimal feedback gain for the unknown system. In this paper, we propose Q-learning-based optimal feedback control taking the delay into consideration. First, we assume that all states can be observed and consider a state feedback controller. The input at the next time is estimated by the current state and the input. Then an optimal feedback gain is provided by sequences of pairs of the state and the input. Next, we show an output feedback controller. If the system is observable, the state can be reconstructed by the last L input and output data, where L is an integer equal to or larger than the observability index of the system. An optimal feedback gain is provided by these sequences. Finally, we apply the proposed adaptive state feedback controller to a quadrotor and show its efficiency by simulation.